Cardinal sine as an activation function for universal classifier training data

ABSTRACT

Cardinal sine function used as an activation function for a hierarchical classifier. Application of a sine function, or a cardinal sine function, for hierarchical classification of a subject within subject matter domains and sub-domains. Hierarchical classification or multi-level classification is improved through use of the cardinal sine function or even standard sine function. Some embodiments of the present invention focus on the usage of cardinal sine function as activation function and how to apply this cardinal sine function for hierarchical classification of a subject. Some embodiments include a technique by which hierarchical classification or multi-level classification can benefit from application of a cardinal sine function.

BACKGROUND

The present invention relates generally to the field of artificialneural networks, and more particularly to activation functions for usein artificial neural networks.

The Wikipedia entry for “artificial neural networks” (ANN) states asfollows: “Artificial neural networks (ANNs), a form of connectionism,are computing systems inspired by the biological neural networks thatconstitute animal brains . . . . Such systems learn (progressivelyimprove performance) to do tasks by considering examples, generallywithout task-specific programming . . . . An ANN is based on acollection of connected units called artificial neurons . . . . Eachconnection (synapse) between neurons can transmit a signal to anotherneuron. The receiving (postsynaptic) neuron can process the signal(s)and then signal downstream neurons connected to it. Neurons may havestate, generally represented by real numbers . . . . Neurons andsynapses may also have a weight that varies as learning proceeds, whichcan increase or decrease the strength of the signal that it sendsdownstream. Further, they may have a threshold such that only if theaggregate signal is below (or above) that level is the downstream signalsent . . . . An (artificial) neural network is a network of simpleelements called neurons, which receive input, change their internalstate (i.e. the activation) according to that input and an activationfunction, and produce output depending on the input and the activation.”

The Wikipedia entry for “activation function” states that the cardinalsine function can be used as an activation function in an ANN.

The Wikipedia entry for “hierarchical classifier” states as follows: “Ahierarchical classifier is a classifier that maps input data intodefined subsumptive output categories. The classification occurs firston a low-level with highly specific pieces of input data. Theclassifications of the individual pieces of data are then combinedsystematically and classified on a higher level iteratively until oneoutput is produced. This final output is the overall classification ofthe data. Depending on application-specific details, this output can beone of a set of pre-defined outputs, one of a set of on-line learnedoutputs, or even a new novel classification that hasn't been seenbefore. Generally, such systems rely on relatively simple individualunits of the hierarchy that have only one universal function to do theclassification. In a sense, these machines rely on the power of thehierarchical structure itself instead of the computational abilities ofthe individual components.”

More generally, a “classifier” is any set of computer software, hardwareand/or firmware for mapping input data into categories, regardless ofwhether the categories are hierarchically organized or not.

SUMMARY

According to an aspect of the present invention, there is a method,computer program product and/or system that performs the followingoperations (not necessarily in the following order): (i) receiving anuntrained hierarchical classifier than includes an artificial neuralnetwork that includes a directed graph model including a plurality ofnodes and a plurality of directed connections among and between thenodes; and (ii) training the untrained hierarchical classifier algorithmto obtain a trained hierarchical classifier using a sinc function as anactivation function.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram view of a first embodiment of a systemaccording to the present invention;

FIG. 2 is a flowchart showing a first embodiment method performed, atleast in part, by the first embodiment system;

FIG. 3 is a block diagram showing a machine logic (for example,software) portion of the first embodiment system;

FIG. 4 is graph showing a sinc function that can be applied in someembodiments of the present invention;

FIG. 5 is a directed graph included in an ANN of an embodiment of thepresent invention; and

FIG. 6 is graph showing a sinc function applied in an embodiment of thepresent invention.

DETAILED DESCRIPTION

This Detailed Description section is divided into the followingsub-sections: (i) The Hardware and Software Environment; (ii) ExampleEmbodiment; (iii) Further Comments and/or Embodiments; and (iv)Definitions.

I. The Hardware and Software Environment

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

An embodiment of a possible hardware and software environment forsoftware and/or methods according to the present invention will now bedescribed in detail with reference to the Figures. FIG. 1 is afunctional block diagram illustrating various portions of networkedcomputers system 100, including: server sub-system 102; clientsub-systems 104, 106, 108, 110, 112; communication network 114; servercomputer 200; communication unit 202; processor set 204; input/output(I/O) interface set 206; memory device 208; persistent storage device210; display device 212; external device set 214; random access memory(RAM) devices 230; cache memory device 232; and program 300.

Sub-system 102 is, in many respects, representative of the variouscomputer sub-system(s) in the present invention. Accordingly, severalportions of sub-system 102 will now be discussed in the followingparagraphs.

Sub-system 102 may be a laptop computer, tablet computer, netbookcomputer, personal computer (PC), a desktop computer, a personal digitalassistant (PDA), a smart phone, or any programmable electronic devicecapable of communicating with the client sub-systems via network 114.Program 300 is a collection of machine readable instructions and/or datathat is used to create, manage and control certain software functionsthat will be discussed in detail, below, in the Example Embodimentsub-section of this Detailed Description section.

Sub-system 102 is capable of communicating with other computersub-systems via network 114. Network 114 can be, for example, a localarea network (LAN), a wide area network (WAN) such as the Internet, or acombination of the two, and can include wired, wireless, or fiber opticconnections. In general, network 114 can be any combination ofconnections and protocols that will support communications betweenserver and client sub-systems.

Sub-system 102 is shown as a block diagram with many double arrows.These double arrows (no separate reference numerals) represent acommunications fabric, which provides communications between variouscomponents of sub-system 102. This communications fabric can beimplemented with any architecture designed for passing data and/orcontrol information between processors (such as microprocessors,communications and network processors, etc.), system memory, peripheraldevices, and any other hardware components within a system. For example,the communications fabric can be implemented, at least in part, with oneor more buses.

Memory 208 and persistent storage 210 are computer-readable storagemedia. In general, memory 208 can include any suitable volatile ornon-volatile computer-readable storage media. It is further noted that,now and/or in the near future: (i) external device(s) 214 may be able tosupply, some or all, memory for sub-system 102; and/or (ii) devicesexternal to sub-system 102 may be able to provide memory for sub-system102.

Program 300 is stored in persistent storage 210 for access and/orexecution by one or more of the respective computer processors 204,usually through one or more memories of memory 208. Persistent storage210: (i) is at least more persistent than a signal in transit; (ii)stores the program (including its soft logic and/or data), on a tangiblemedium (such as magnetic or optical domains); and (iii) is substantiallyless persistent than permanent storage. Alternatively, data storage maybe more persistent and/or permanent than the type of storage provided bypersistent storage 210.

Program 300 may include both machine readable and performableinstructions and/or substantive data (that is, the type of data storedin a database). In this particular embodiment, persistent storage 210includes a magnetic hard disk drive. To name some possible variations,persistent storage 210 may include a solid state hard drive, asemiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer-readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 210 may also be removable. Forexample, a removable hard drive may be used for persistent storage 210.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer-readable storage medium that is also part of persistent storage210.

Communications unit 202, in these examples, provides for communicationswith other data processing systems or devices external to sub-system102. In these examples, communications unit 202 includes one or morenetwork interface cards. Communications unit 202 may providecommunications through the use of either or both physical and wirelesscommunications links. Any software modules discussed herein may bedownloaded to a persistent storage device (such as persistent storagedevice 210) through a communications unit (such as communications unit202).

I/O interface set 206 allows for input and output of data with otherdevices that may be connected locally in data communication with servercomputer 200. For example, I/O interface set 206 provides a connectionto external device set 214. External device set 214 will typicallyinclude devices such as a keyboard, keypad, a touch screen, and/or someother suitable input device. External device set 214 can also includeportable computer-readable storage media such as, for example, thumbdrives, portable optical or magnetic disks, and memory cards. Softwareand data used to practice embodiments of the present invention, forexample, program 300, can be stored on such portable computer-readablestorage media. In these embodiments, the relevant software may (or maynot) be loaded, in whole or in part, onto persistent storage device 210via I/O interface set 206. I/O interface set 206 also connects in datacommunication with display device 212.

Display device 212 provides a mechanism to display data to a user andmay be, for example, a computer monitor or a smart phone display screen.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

II. Example Embodiment

FIG. 2 shows flowchart 250 depicting a method according to the presentinvention. FIG. 3 shows program 300 for performing at least some of themethod operations of flowchart 250. This method and associated softwarewill now be discussed, over the course of the following paragraphs, withextensive reference to FIG. 2 (for the method operation blocks) and FIG.3 (for the software blocks).

Processing begins at operation S255, where untrained hierarchicalclassifier 302 is received. This untrained hierarchical classifierincludes artificial neural network 304 that includes a directed graphmodel 306 including multiple neural nodes 308 and multiple directedconnections 310 among and between the nodes.

Processing proceeds to S260 where training module (“mod”) 320 trains theuntrained hierarchical classifier algorithm to obtain a trainedhierarchical classifier 302 using a sinc function as an activationfunction 322 and multiple sets of training data 324 that arerespectively selected to range across a hierarchical category at somelevel of the hierarchy. For example, a wide ranging training data set isused to train the highest level of the hierarchy. Multiple intermediatelevel training data sets respectively train hierarchical categories at anext lower level of the hierarchy. Multiple low level training data setsare used to respectively train for making identifications at even lowerlevels of the hierarchy.

Processing proceeds to operation S265, where classify input data mod 330applies trained hierarchical classifier to a set of input data (receivedfrom client 104 through communication network 114 (see FIG. 1)) todetermine multiple hierarchical categories to which the input data setbelongs. In this case, the input data is a file with audio representingthe sound of a malfunctioning jet engine. In this example, the classifyinput data mod classifies the sound as belonging to three hierarchicalcategories existing to three levels of a hierarchy of sounds as follows:(i) highest level hierarchical category=machinery sounds; (ii)intermediate level hierarchical category=vehicular machinery sounds; and(iii) low level category=aircraft engine sounds. Methods of the traininghierarchical classifiers, using with the sinc activation function, willbe discussed in more detail in the following sub-section of thisDetailed Description section.

III. Further Comments and/or Embodiments

Some embodiments of the present invention recognize the following facts,potential problems and/or potential areas for improvement with respectto the current state of the art: (i) Artificial Neural Network (ANN) isa known computational technique to classify a pattern (this is,determine whether the pattern belongs in a particular category (in anon-hierarchical classifier context) or in a multiple hierarchicallyrelated categories in a hierarchical classifier context—for example apattern of data classified in the category “dog” would also beclassified in the supercategory of “mammal”); (ii) ANN is used inpattern recognizing and supervised machine learning; (iii) an ANN can berepresented as a weighted graph where: (a) nodes are responsible for amathematical calculation, and (b) edges are the inputs based on whichcalculations are executed in nodes; and/or (iv) the nodes (except theinput layer) operate somewhat similarly to biological neurons which are:(a) activated only when a certain threshold is crossed, or, otherwise(b) keep dormant.

In currently conventional classifier implementations, activation isachieved through a set of mathematical equations in ANN as follows:y=w*x+by _(output) =f(y)where w is the aforementioned weight of the input edge, x is the inputcoming from the previous layer or input layer, b is the bias, y_(output)is the output of a node and f(y) is an activation function.

Currently conventional ANNs typically use sigmoid functions asactivation functions, Rectified Linear Unit etc. The sigmoid functionsdo not have any oscillation and have only one global extremum but nolocal extremum when used as an activation function to train the ANN.Whereas these networks are perfectly fine for training a ANN (ArtificialNeural Network) that learns to classify a single type of object. Say, ifit is desired to classify specific breed of dogs from a picture of dog,this type of network is fine but if a currently conventional classifierprovide a picture of any other animal (say, a picture of an elephant) asthe input to that network, the currently conventional classifier: (i)will try to match that picture against the various breeds of dog definedduring training of the classifier; and (ii) will likely produce anerroneous classification or a classification of limited informationalvalue (for example, a classification that an elephant is simplysomething other than a dog).

At least some currently conventional classifiers can't say “NO” to input(for example, these currently conventional classifiers are notprogrammed to have the capacity to classify an elephant simply in a“non-dog category”). This means that these currently conventionalclassifiers can't take any decision as it can't decide if the inputimage lies within the intended domain of classification. Moreover, evenwith the pictures from the classification domain taken as input,currently conventional classifiers can only classify the breed of thedog in our example but not the other, higher hierarchical levels thatare also applicable to the picture of dog, like the dog's genus, family,suborder, order, etc. However, some embodiments of the present inventionrecognize that this kind of hierarchical categorization is quite usefulfor a more demanding classification scenario.

Consider a trained classifier (sometimes also herein referred to as a“model”) that has been trained only to determine breed of a dog from aphotograph of the dog. In this example situation, a user inputs to thetrained model a picture of a strange animal that he has not seen earlier(for the simplicity, suppose the user doesn't know any animals otherthan dog) and that the picture input by the user certainly is not a dog.The user wants to get some information about the animal because, lateron, the user can have more specialized search performed on theunidentified animal in his picture. If a conventional neural network isused, it would match the image with a particular breed of dog to whichit found the maximum similarity. However, some neural network modelsthat are trained according to inventive methods of the present inventionwill give the user information up to the name of the animal as towhether it is a cat, mongoose, elephant, etc.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) a designof an artificial neural network using cardinal sine activation functionthat will hierarchically classify a problem input from broaderclassification to a more specialized one; and/or (ii) a hierarchicalclassifier including a neural network. As specific example of anembodiment of item (ii) in the preceding list: a hierarchical classifierthat can classify an image of an Indian Pariah dog as belonging to allof the following hierarchically related categories: animal, vertebrate,mammal, canine and Indian Pariah. In this example, the classificationdomain can be set as Animal Kingdom, with the result that thisclassifier won't classify further any image if it is not an animal.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) use ofthe sine function and/or cardinal sine (“sinc”) function as anactivation function for the ANN of a classifier (for example, ahierarchical classifier); (ii) use of a sinc curve with multiple localminimum where the amplitude of the curve decreases with increasingperiod of the curve in each direction; (iii) classifier with an ANNmodeled after sinc function that can perform more specialized patternrecognition as the period of the curve is increased; and/or (iv) theactivation function affects significantly in error reducing of costfunction of the ANN.

In some embodiments of classifiers according to the present invention,the training set for the network will be divided into multiple sets.First set will be introduced to the network with weights initializedafter a specific manner described below. The weights will stabilize forthe root of the hierarchy intended for the ANN. Then the second trainingset will be introduced to the network. Weights in the network for thesecond training set have to be initialized after a specific mannerdescribed below. The second training set is more specialized than thefirst one. The second training set will stabilize the weights of thenetwork again for the child hierarchy. This way, more specializedtraining set will be added as much as specialization the network isintended. The user/implementer of the network has to remember theweights after each training set.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) if theinput is not intended for the network, the input will be rejected midwayinstead of producing misleading output; (ii) a universal/hierarchicalclassifier can be produced using this proposed type and theoreticallythis network will be able to recognize every pattern in universe inhierarchical manner; (iii) the network will be able to guess the outputto some extent if the network is not built to classify the input (forexample, if the input is of elephant but not the specific breed of dogs,this network will at least classify the input as elephant; (iv) whateveradvancement is made in training the model in parallel, it requiresmerging the finding in each parallel training to other ones—much likeMap-Reduce techniques; (v) these merging operations are extra overheadwhich is completely removed in some embodiments of classifiers accordingto the present invention; (vi) that large training set of conventionalnetwork is divided in multiple training sets with each subsequent setsare more specialized than its previous one; (vii) each training setdeduces different weights of the network model; and/or (viii) as theweights are not dependent on the results of other training set, thetrainings can be done in complete parallel with no requirements formerging.

An embodiment of a hierarchical classifier according to the presentinvention will be described in the following paragraphs.

Cardinal Sine or Sinc curve:

f(x)=1 for x=0

f(x)=sin(x)/x for x≠0

The above sinc curve has domain of [−6 π, 6 π]. Modifying the function alittle, it can be formulated as follows (this formulation is sometimesreferred to as a “Truncated Cardinal Sine” or “Truncated Sinc”):

f(x)=1 for x=0

f(x)=sin(x)/x for x>0

f(x)=1 for x<0

As shown in graph 400 of FIG. 4, the left part of the curve (from x=0 inthe graph) will be always at 1. Then, the steepest slope appears firstand as we go right, the curve's oscillation reduces. The classifierembodiment now under discussion applies this property of TruncatedCardinal Sine curve to the proposed design of its ANN component.

In the embodiment now under discussion, the aim is to find the globalminimum for one training set in case of most of the conventional neuralnetwork. In this model ANN: (i) there are multiple training sets; (ii)local minimum, instead of global minimum, is found for each trainingset; and (iii) for each training set, the weight of the edges connectingthe nodes is initiated after a fashion so that each node output alwaysresides in a separate concave portions (trough) of the curve for eachtraining set. In the embodiment now under discussion, the foregoing isthe main logic behind how Truncated Cardinal Sine function acts ashierarchical classifier.

The overall design of the ANN for the classifier embodiment now underdiscussion, which uses Truncated Cardinal Sine function as an activationfunction, will now be discussed. The ANN is trained with multipletraining sets where each training set will be more specialized (that is,will represent a lower level of the hierarchy of classificationcategories) one than the previous training set. In this hierarchicalclassifier embodiment, the hierarchical classifier will accept anypicture of any vertebrate animal and is only used to recognize aparticular breed of dogs (for example, Greyhound). The hierarchicalclassifier categorizes the proposed design of network as third degree,as will be further discussed, below. From the perspective of the enduser of the hierarchical classifier, there are three degrees as follows:(i) at first the network will identify the hierarchical category of“mammal” from the input picture of the vertebrate animal (that is,“mammal” is the first degree (or first level) hierarchical category);(ii) then the hierarchical classifier identifies the hierarchicalcategory of “dog” out of the hierarchical child categories of the“mammal” category (that is, “dog” is the second degree (or second level)hierarchical category); and (iii) then the hierarchical classifieridentifies the hierarchical category of “Greyhound” as the appropriatechild hierarchical category of “dog” (that is, “Greyhound” is the thirddegree (or third level) hierarchical category). As it is total threelevel of recognition, it is named “third degree.” This embodiment canincrease the number of degrees of the network to provide as muchspecificity (or generalization) as demanded by a particular application.

Directed graph 500 of FIG. 5, shows an example of an ANN for use in thehierarchical classifier embodiment now under discussion. The ANNconsists of two input neurons, three hidden neurons and two outputneurons. Each neuron in a layer is connected to that of previous layerwith a connection that carries a specific weight. Input neurons havevalues equal to the input value they are provided with. In this case, I₁has value equal to that of Input₁ and I₂ has value equal to that ofInput₂.

The ANN of the hierarchical classifier embodiment under discussion: (i)is provided with a first training data set; and (ii) in response,updates the weights of the directed graph of the ANN in a manner thatwill now be discussed in the following paragraphs.

The Forward Pass will now be discussed. The total net input to eachhidden layer neuron is determined by: (i) squashing the total net inputusing Positive Cardinal Sine function; and (ii) hence, obtaining thevalue of each hidden layer neuron.

Considering an example with a total net input of H₁ neuron as net_(h1)net_(h1)=W₁*I₁+W₄*I₂ +b ₁where, b₁ is the bias added for the hidden layer. In this proposed modelof hierarchical classifier, certain considerations are applied inselecting the weights. In conventional ANNs, the weights are initializedwith a random number, but the hierarchical classifier now underdiscussion determines the weights carefully so that net_(h1) liesbetween ^(π) to 2^(π) for the first training set. W₁ and W₄ arecalculated in a way that will be discussed in the following paragraphs.

Step 1: Take W₄*I₂ term out of consideration for now. This yields atemporary equation as follows:net_(H1)=W₁*I₁ +b ₁Or, net_(H1) −b ₁=W₁*I₁Here, b₁ is a fixed value for a particular hidden layer. This meansthat:^(Π≤net) _(H1) ^(≤)2^(π)It then follows that:^(π) −b1^(≤)W₁*I₁ ^(≤)2^(π) −b ₁The minimum value of W₁ is:(^(π) −b ₁)/Max(I₁)and maximum value of W₁ is:(2^(π) −b1)/Min(I₁)The value of W₁ is initialized with any value within this range. Moreintuitive consideration may increase this range of values that W₁ cantake.

Step 2: W₄ is calculated and brings up the original equation of:net_(H1)=W₁*I₁+W₄*I₂ +b ₁This means that:^(π≤)net_(H1)≤2^(π)Or, ^(π) −b1≤W₁*I₁+W₄*I₂ ^(≤)2^(π) −b ₁Or,(^(π) −b1−W₁*I₁)^(≤W) ₄*I₂ ^(≤)(2^(π) −b ₁−W₁*I₁)So, W₄ lies between:((^(π) −b ₁−W₁*Max(I₁))/Max(I₂)) to ((2^(π) −b ₁−W₁*Min(I₁))/Min(I₂))All of these careful considerations for the weights are taken so thatnet_(H1) could lie on the curve in the region enclosed by the dottedrectangle shown in graph 600 of FIG. 6.

The net_(H1) is then squashed using the positive sinc function to getthe output of H₁ neuron as follows:output_(H1)=sin(net_(H1))/net_(H1)The above mentioned steps are repeated for all of the other weights inhidden layer 1.

Weights and neuron values in output layer (that is, from hidden neuronsto output neurons) are now calculated. Here is the output for Output₁:net_(O1)=W₇*output_(H1)+W₉*output_(H2)+W₁₁*output_(H3) +b ₂The weights in this layer are determined in a manner similar to thatexplained in connection with the previous layer using the followinginequality:^(π≤)net_(O1)≤2^(π)And then the output of O₁ is determined as follows:output_(O1)=sin(net_(O1))/net_(O1)Then carrying out the same process for O₂:output_(O2)=sin(net_(O2))/net_(O2)A softmax function may be used in this layer to squish the outputswithin [0, 1].

Calculating the Total Error will now be discussed. The error iscalculated for each output neuron using the squared error function andthen summed to get the total error as follows:Σ_(total)=Σ½(target−output)²

The Backwards Pass will now be discussed. Backpropagation is applied toupdate each of the weights in the network so that they cause the actualoutput to be closer to the target output, thereby minimizing the errorfor each output neuron and the network as a whole. Consider W₇. Todetermine how much a change in W₇ affects the total error:aka ^(δ) E _(total)/^(δ)W₇^(δ) E _(total)/^(δ)W₇=^(δ) E_(total)/^(δ)output^(O1)*^(δ)output^(O1)/^(δnet) _(O1)*^(δnet)_(O1)/^(δ)W₇This is the standard method of reducing error in currently conventionalANNs, and, therefore, those of skill in the art will understand that theweights of the edges of the directed graph of the ANN can be stabilizedso that the ANN can recognize any pattern with a relatively high levelof reliability.

This embodiment of a hierarchical classifier provides multiple trainingsets, as mentioned earlier. The first training data set, in this exampleof recognizing a data pattern as belonging to the “Greyhound”hierarchical child category of the “dog” category, consists only ofpictures of vertebrate animals spanning over the entire category of“vertebrate” (that is, the training data set includes images of birds,images of reptiles, images of dogs, images of cats, images of humans,etc.). The network weights will stabilize according to the abovementioned technique. The second training data set consists of images ofmammals, like humans, dogs, cats, tigers, etc. Before this mammaliantraining data set is applied, new weights are initiated for each of theedges of the directed graph of the ANN, similar to what was done for thevertebrate training data set. In this case, limits for net_(H1) will be:3^(π≤net) _(H1) ^(≤)4^(π)Minimum and maximum limit of net_(H1) is increased by exactly 2^(π).Similarly, minimum and maximum limit of netO1 is increased by exactly2^(π). And so on for the next training sets.

Now it will be explained why the ANN of this embodiment is characterizedas “third degree,” with reference to the sinc curves of FIG. 4 and FIG.6. The graphs of FIGS. 4 and/or 6 show that if one proceeds through theabove mentioned design of the ANN of this embodiment, then the localminima of all the vertebrates (that is, the vertebrate training dataset) will reside in the region marked in the graph 600 of FIG. 6. Theinput value of the function increases for the next mammal training dataset and the weights of the edges of the directed graph are initiatedbased on that. This means that the mammal training data set will residein the region marked by mammal, and so on through the increasing degrees(that is, the increasing hierarchical levels of categories implementedin the hierarchical classifier).

Only three concave parts (troughs) of the sinc function are occupied forthis particular network, so the ANN of this hierarchical classifierembodiment is characterized as being of the third degree. Alternatively,ANNs of higher degrees can be generated that can handle a wider ofinputs to be classified by the classifier. This is how an end user canuse this model to categorize her picture. It is helpful to remember theweights of the network after each training data set. For example, whenan end user provides a picture of, say, a monkey, the network will usethat set of weights eventuated from the first training set. The networkwill, with relatively high reliability, classify the monkey image in thehierarchical category of “mammals.” The end user should, therefore,apply to the edges of the directed graph of the ANN that set of weightswhich determined after the second training set. The ANN will notreliably classify the monkey image in the next level hierarchicalcategory of “dogs,” so the end user should stop the network from furtherprocessing.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) the ANNcan act as a hierarchical classifier; (ii) the ANN can be called a smartneural network which classifies the input more efficiently andeffectively than currently conventional neural networks; (iii) as themodel can be trained with multiple training sets; (iv) each training settrains the models with different weights; (v) the training can happen inparallel without the need of any extra calculation typically required incurrently conventional ANNs (that is, parallel training requires a lotof extra calculation to merge the weights deduced from each paralleltraining); (vi) this design can handle a larger larger variety of inputthan current conventional ANNs and classify them smartly; and/or (vii)this design can find its application in exploring new patterns,classifying patterns from a large variety of inputs.

An embodiment of a method according to the present invention includesthe following operations: (i) initiate the weights and input matrices;(ii) produce the training sets where each set is more specialized thanthe previous one; (iii) assign the weights in the network in the abovementioned way; (iv) produce the values from the set as input to thenetwork; (v) calculate the inner neuron values using truncated cardinalsine function; (vi) calculate the output values of neurons and the errorfor each output neuron using the squared error function and sum up themto get the total error; (vii) apply backpropagation algorithm until thenetwork stabilizes; (viii) after stabilizing the network, save theweights for each training set; and (ix) if there is one more trainingset, go to step (ii) or else return.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i)hierarchical classification of the input subject; (ii) completelyparallel training of the network model with no requirement to merge theresults found in each training; (iii) smart neural network that won'tclassify the input object if it is not intended to be classified by thenetwork but provide a great deal of hierarchical information of thatobject; (iv) a Hierarchical Classifier or Universal Classifier orMulti-level Classifier; (v) a classifier that takes more than one typeof input and produces more than one hierarchical category leveldetermination based on the inference drawn on the input subject(s); (vi)if a new subject is introduced as input to the classifier, erroneousmatching against the closest similar object is avoided so that morereliable and useful information is obtained from the inferences ofclassifiers according to the present invention; (vii) subject will beclassified hierarchically from its generalized category to more aspecialized category; (viii) able to extract more information from asubject than currently conventional classifiers; (ix) if the inputsubject doesn't match to any desired output, some information about thetype of the input subject still can be obtained using this classifier;and/or (x) the name of classifiers according to some embodiments of thepresent invention can be accurately termed as “Universal Classifiers”because the classifier can extract information about the input subjecttill a certain extent even if it is completely unknown to theclassifier.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) use ofCardinal Sine as an activation function in the universal classifier;(ii) use of Sine curves as an activation function in an ANN of aclassifier (as sine curves exhibit oscillation and hence, multiple localminimum; (iii) reduces chances that a local minimum will be erroneouslyconsidered as global minimum while calculating the weights of edges of adirected graph of a classifier's ANN; (iv) net output can be boundedwithin one trough of the sine curve, but, if only one trough of thecurve is used to train all of the training data, then that trough can beeasily saturated and so, some embodiments divide the training data intomultiple sets; and/or (v) use of multiple training data sets that: (a)start with a wider range of subject matter to train on higherhierarchical level(s) of classification, (b) continue training withintermediate range of subject matter data sets to train on intermediatehierarchical level(s) of classification, and (c) further continuetraining with specific ranges of subject matter data sets to train onlower hierarchical level(s) of classification.

As an example of item (v) in the list of the preceding paragraph, incase of classifying a particular breed of dog, our training set can bedivided as Vertebrates→Mammals→Quadruped Animals→Canines→All the breedof the dogs. Now, for such a trained model, information can be reliablygathered about the input subject as far as the subject progress througheach hierarchy. In some embodiments, cardinal Sine or Sinc curve is usedas an activation function because of the above explained progressionfrom generalized training data sets to more specific training data sets.Similarities between the patterns in each training set will increase. Assinc curve has this property of gradual decreasing troughs and, hence,gradually decreasing length of the curve, finding global minimum in eachtrough will take less time as processing goes forward within thehierarchical structure.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) acomplete parallel training that does not require to merge the results;(ii) whatever advancement is made in training the model in parallel, itrequires merging the finding in each parallel training to otherones—much like Map-Reduce techniques; (iii) avoids these mergingoperations, and associated extra overhead; (iii) each training setdeduce different weights of the network model; and/or (iv) as theweights are not dependent on the results of other training set, thetrainings can be done in complete parallel with no requirements ofmerging.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) usecardinal sine or sinc function as an activation function for theartificial neural network; (ii) as sinc curve has multiple local minimumand the amplitude of the curve decreases with increasing period of thecurve in each direction, the network modeled after sinc function hasthis particular feature of specializing in pattern recognition as theperiod of the curve is increased; (iii) the activation function affectssignificantly in error reducing of cost function of the ANN; (iv) inthis mechanism, the training set for the network will be divided intomultiple sets; (v) first set will stabilize the weights for use inmaking the highest hierarchical level classification intended for theANN; (vi) the second training set will be introduced to the network,such that weights in the network for the second training set have to beinitialized after a specific manner described, below; (vii) the secondtraining set is more specialized than the first one; (viii) the secondtraining set will stabilize the weights of the network again; (ix) thisway, more specialized training set will be added as much asspecialization the network is intended; and/or (x) the user/implementerof the network should to remember the weights after each training set.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) if theinput is not intended for the network, the input will be rejected midwayinstead of producing misleading output; (ii) universal classifier can beproduced using this proposed type and theoretically this network will beable to recognize every pattern in universe; (iii) the network will beable to guess the output to some extent if the network is not built forthe input (that is, if the input is of elephant but not the specificbreed of dogs, this network will at least classify the input up to themammal); and/or (iv) performs hierarchical classification with the useof cardinal sine function in a special technique.

Some embodiments of the present invention may include one, or more, ofthe following features, characteristics and/or advantages: (i) providesboth of the following: (a) a basic design of hierarchical classifierthat implements multiple classifiers from generalized (coarse) tospecialized (detailed) ones, and (b) usage of cardinal sine function (orregular sine function) as activation function; (ii) uses cardinal sinefunction in a unique way to solve the problem of hierarchicalclassification; (iii) bounded region of the weights in the neuralnetwork is updated before the starting of each training; (iv)backpropagation algorithm does not have to explore a vast region forglobal minimum but only a small bounded region; (v) avoids computationrelative to classifiers where backpropagation algorithms need to findthe global minimum again from scratch across the vast region same forcoarse to detailed classification; and/or (vi) reduces the boundedregion as progressed further along the curve wherein this property isnot available for sine curve or cosine curve.

IV. Definitions

Present invention: should not be taken as an absolute indication thatthe subject matter described by the term “present invention” is coveredby either the claims as they are filed, or by the claims that mayeventually issue after patent prosecution; while the term “presentinvention” is used to help the reader to get a general feel for whichdisclosures herein are believed to potentially be new, thisunderstanding, as indicated by use of the term “present invention,” istentative and provisional and subject to change over the course ofpatent prosecution as relevant information is developed and as theclaims are potentially amended.

Embodiment: see definition of “present invention” above—similar cautionsapply to the term “embodiment.”

and/or: inclusive or; for example, A, B “and/or” C means that at leastone of A or B or C is true and applicable.

Including/include/includes: unless otherwise explicitly noted, means“including but not necessarily limited to.”

Module/Sub-Module: any set of hardware, firmware and/or software thatoperatively works to do some kind of function, without regard to whetherthe module is: (i) in a single local proximity; (ii) distributed over awide area; (iii) in a single proximity within a larger piece of softwarecode; (iv) located within a single piece of software code; (v) locatedin a single storage device, memory or medium; (vi) mechanicallyconnected; (vii) electrically connected; and/or (viii) connected in datacommunication.

Computer: any device with significant data processing and/or machinereadable instruction reading capabilities including, but not limited to:desktop computers, mainframe computers, laptop computers,field-programmable gate array (FPGA) based devices, smart phones,personal digital assistants (PDAs), body-mounted or inserted computers,embedded device style computers, application-specific integrated circuit(ASIC) based devices.

What is claimed is:
 1. A computer-implemented method comprising:receiving an untrained hierarchical classifier than includes anartificial neural network that includes a directed graph model includinga plurality of nodes and a plurality of directed connections among andbetween the nodes; and training the untrained hierarchical classifieralgorithm to obtain a trained hierarchical classifier using a sincfunction as an activation function; wherein: the sinc function is atruncated sinc function defining a plurality of troughs; and each troughof the plurality of troughs corresponds to one hierarchical level of ahierarchy of categories to be identified by the trained hierarchicalclassifier.
 2. The method of claim 1 further comprising: receiving afirst input data set; and applying the trained hierarchical classifierto the first input data set to identify a plurality of hierarchicalcategories, respectively at different hierarchical levels, to which thefirst input data set belongs.
 3. The method of claim 1 wherein thetraining includes the use of a plurality of training data setsincluding: a first training data set including training data rangingacross a range defined by a highest level category of a hierarchy ofcategories; a second training data set including training data rangingacross a range defined by a first intermediate level category of thehierarchy of categories, with the highest level category being inclusiveof the first intermediate level category; and a third training data setincluding training data ranging across a range defined by a first lowlevel category of the hierarchy of categories, with the firstintermediate level category being inclusive of the first low levelcategory.
 4. The method of claim 3 further comprising: receiving a firstinput data set; and applying the trained hierarchical classifier to thefirst input data set to identify the first input data set as indicativeof subject matter belonging to all of the following hierarchicalcategories: the highest level category, the first intermediate levelcategory and the first low level category.
 5. The method of claim 1wherein the training includes: application of a backpropagationalgorithm that explores only a bounded region defined by the sincfunction.
 6. A computer program product comprising: a non-transitorymachine readable storage device; and computer code stored on thenon-transitory machine readable storage device, with the computer codeincluding instructions for causing a processor(s) set to performoperations including the following: receiving an untrained hierarchicalclassifier than includes an artificial neural network that includes adirected graph model including a plurality of nodes and a plurality ofdirected connections among and between the nodes, and training theuntrained hierarchical classifier algorithm to obtain a trainedhierarchical classifier using a sinc function as an activation function;wherein: the sinc function is a truncated sinc function defining aplurality of troughs; and each trough of the plurality of troughscorresponds to one hierarchical level of a hierarchy of categories to beidentified by the trained hierarchical classifier.
 7. The computerprogram product of claim 6 wherein the computer code further includesinstructions for causing the processor(s) set to perform the followingoperations: receiving a first input data set; and applying the trainedhierarchical classifier to the first input data set to identify aplurality of hierarchical categories, respectively at differenthierarchical levels, to which the first input data set belongs.
 8. Thecomputer program product of claim 6 wherein the training includes theuse of a plurality of training data sets including: a first trainingdata set including training data ranging across a range defined by ahighest level category of a hierarchy of categories; a second trainingdata set including training data ranging across a range defined by afirst intermediate level category of the hierarchy of categories, withthe highest level category being inclusive of the first intermediatelevel category; and a third training data set including training dataranging across a range defined by a first low level category of thehierarchy of categories, with the first intermediate level categorybeing inclusive of the first low level category.
 9. The computer programproduct of claim 8 wherein the computer code further includesinstructions for causing the processor(s) set to perform the followingoperations: receiving a first input data set; and applying the trainedhierarchical classifier to the first input data set to identify thefirst input data set as indicative of subject matter belonging to all ofthe following hierarchical categories: the highest level category, thefirst intermediate level category and the first low level category. 10.The computer program product of claim 6 wherein the training includes:application of a backpropagation algorithm that explores only a boundedregion defined by the sinc function.
 11. A computer system comprising: aprocessor(s) set; a non-transitory machine readable storage device; andcomputer code stored on the non-transitory machine readable storagedevice, with the computer code including instructions for causing theprocessor(s) set to perform operations including the following:receiving an untrained hierarchical classifier than includes anartificial neural network that includes a directed graph model includinga plurality of nodes and a plurality of directed connections among andbetween the nodes, and training the untrained hierarchical classifieralgorithm to obtain a trained hierarchical classifier using a sincfunction as an activation function; wherein: the sinc function is atruncated sinc function defining a plurality of troughs; and each troughof the plurality of troughs corresponds to one hierarchical level of ahierarchy of categories to be identified by the trained hierarchicalclassifier.
 12. The computer system of claim 11 wherein the computercode further includes instructions for causing the processor(s) set toperform the following operations: receiving a first input data set; andapplying the trained hierarchical classifier to the first input data setto identify a plurality of hierarchical categories, respectively atdifferent hierarchical levels, to which the first input data setbelongs.
 13. The computer system of claim 11 wherein the trainingincludes the use of a plurality of training data sets including: a firsttraining data set including training data ranging across a range definedby a highest level category of a hierarchy of categories; a secondtraining data set including training data ranging across a range definedby a first intermediate level category of the hierarchy of categories,with the highest level category being inclusive of the firstintermediate level category; and a third training data set includingtraining data ranging across a range defined by a first low levelcategory of the hierarchy of categories, with the first intermediatelevel category being inclusive of the first low level category.
 14. Thecomputer system of claim 13 wherein the computer code further includesinstructions for causing the processor(s) set to perform the followingoperations: receiving a first input data set; and applying the trainedhierarchical classifier to the first input data set to identify thefirst input data set as indicative of subject matter belonging to all ofthe following hierarchical categories: the highest level category, thefirst intermediate level category and the first low level category. 15.The computer system of claim 11 wherein the training includes:application of a backpropagation algorithm that explores only a boundedregion defined by the sinc function.